Enhanced collection of training data for machine learning to improve worksite safety and operations

ABSTRACT

Systems and methods for enhanced collection of training data for machine learning to improve worksite safety and operations. One embodiment is a system with an interface to receive first evaluations of a first scene from a group of trainees, the first scene belonging to safety curriculum content and depicting a worksite with a known hazard that is associated in memory with a hazard profile. The system includes a controller to determine a trusted subgroup of the trainees that correctly identified the known hazard in the first scene. The interface receives second evaluations of a second scene from the trusted subgroup of the trainees that depicts the worksite with an unknown hazard. The controller trains a machine learning function based on the second evaluations from the trusted subgroup of the trainees for automatic identification of hazard indications in the second scene depicting the worksite with the unknown hazard.

FIELD

The disclosure relates to the field of worksite operation and safety.

BACKGROUND

A large organization or company may have employees take safety trainingcourses to certify their ability to identify and prevent potentiallydangerous conditions that may arise in their line of work.Unfortunately, the safety training that is offered may become stagnant,limited, or not comprehensive due to the expense involved withcontinually updating or creating new safety curriculum content as newworksite issues arise. Additionally, even with mandatory and/or periodicsafety courses, the potential for human error at worksites remains apossibility. Companies therefore seek to provide the most comprehensivesafety training possible as well as improved worksite safety systems tohelp detect and prevent safety hazards.

Machine learning is a developing field that promises automation in manyapplications. However, machine learning generally performs best inapplications in which large amounts of training data is available forinput. It is prohibitively difficult for companies to implement machinelearning for improving a worksite safety system because it would involvea heavy cost in manual work to collect, filter, and classify a largeenough amount of training data to be useful for input into the machinelearning. Thus, the conventional approach to detecting worksite safetyrelies on workers physically monitoring the worksite, which is manuallyintensive and expensive. Moreover, the conventional approach toresponding to a potentially unsafe event may be limited by human recallof training content that has not been comprehensively updated.

SUMMARY

Embodiments described herein provide enhanced collection of trainingdata for machine learning to improve worksite safety and operations.Techniques described herein leverage an organization's safety trainingcurriculum process for their personnel to generate improved datatraining input for a machine learning system. Human evaluators taking asafety class are shown a mixed pattern of situations that are both knownto be hazardous and unknown to be hazardous. The example safetyscenarios are refined by applying weighted trusts to the humanevaluators identifying potential and real safety threats anddescribing/qualifying those risks as they take safety training courses.Advantageously, these refined examples can then be reused to improve thesafety course content. Additionally, the refined examples are input to amachine learning system to improve its output in automatically detectingand responding to safety hazards and risks in real time.

Embodiments described herein further provide a gamification system toincentivize the collection of user-classified data for a machinelearning function. The gamification system awards points to users forcorrectly capturing an object such as a machine or person performing atarget action. The gamification system thus establishes users which haverelative higher ratings or levels of trust in accuratelyassessing/capturing situations presented in an environment. A user'sscore changes the weighting consideration of their input into machinelearning relative to their peers, thus refining the input training datathrough the gamification system.

One embodiment is a worksite safety system that includes an interfaceconfigured to receive first evaluations of a first scene from a group oftrainees, the first scene belonging to safety curriculum content anddepicting a worksite with a known hazard that is associated in memorywith a hazard profile. The worksite safety system also includes a safetytraining controller configured to determine a trusted subgroup of thetrainees that correctly identified the known hazard in the first scenebased on a match between the first evaluations and the hazard profile.The interface is further configured to receive second evaluations of asecond scene from the trusted subgroup of the trainees, the second scenedepicting the worksite with an unknown hazard. The safety trainingcontroller is further configured to train a machine learning functionbased on the second evaluations from the trusted subgroup of thetrainees for automatic identification of hazard indications in thesecond scene depicting the worksite with the unknown hazard.

Another embodiment is a method of training a machine learning functionfor worksite safety. The method includes receiving first evaluations ofa first scene from a group of the trainees, the first scene belonging tosafety curriculum content and depicting a worksite with a known hazardthat is associated in memory with a hazard profile. The method alsoincludes determining a trusted subgroup of the trainees that correctlyidentified the known hazard in the first scene based on a match betweenthe first evaluations and the hazard profile, and receiving secondevaluations of a second scene from the trusted subgroup of the trainees,the second scene depicting the worksite with an unknown hazard. Themethod also includes training the machine learning function based on thesecond evaluations from the trusted subgroup of the trainees forautomatic identification of hazard indications in the second scenedepicting the worksite with the unknown hazard.

Yet another embodiment is non-transitory computer readable mediumembodying programmed instructions executed by a processor, wherein theinstructions direct the processor to perform a method of training amachine learning function for worksite safety. The method includesreceiving first evaluations of a first scene from a group of thetrainees, the first scene belonging to safety curriculum content anddepicting a worksite with a known hazard that is associated in memorywith a hazard profile. The method also includes determining a trustedsubgroup of the trainees that correctly identified the known hazard inthe first scene based on a match between the first evaluations and thehazard profile, and receiving second evaluations of a second scene fromthe trusted subgroup of the trainees, the second scene depicting theworksite with an unknown hazard. The method also includes training themachine learning function based on the second evaluations from thetrusted subgroup of the trainees for automatic identification of hazardindications in the second scene depicting the worksite with the unknownhazard.

Other example embodiments (e.g., methods and computer-readable mediarelating to the foregoing embodiments) may be described below. Thefeatures, functions, and advantages that have been discussed can beachieved independently in various embodiments or may be combined in yetother embodiments further details of which can be seen with reference tothe following description and drawings.

DESCRIPTION OF THE DRAWINGS

Some embodiments of the present disclosure are now described, by way ofexample only, and with reference to the accompanying drawings. The samereference number represents the same element or the same type of elementon all drawings.

FIG. 1 illustrates a worksite environment in an illustrative embodiment.

FIG. 2 is a flowchart illustrating a method of refining training datafor a machine learning function to enhance worksite safety in anillustrative embodiment.

FIG. 3 is a flowchart illustrating a method of applying refined trainingdata to a machine learning function for worksite safety in anillustrative embodiment.

FIG. 4 is a flowchart illustrating a method of refining training datafor a machine learning function to enhance worksite safety in anotherillustrative embodiment.

FIG. 5 is a flowchart illustrating a method of modifying safetycurriculum content in an illustrative embodiment.

FIG. 6 is a block diagram of a training system in an illustrativeembodiment.

FIG. 7 is a flowchart illustrating a method of collecting training datafor a machine learning function to enhance worksite operations in anillustrative embodiment.

FIG. 8 is a flowchart illustrating a method for determining to applysensor data to a machine learning function in an illustrativeembodiment.

FIG. 9 is a flowchart illustrating a method for determining to applysensor data to a machine learning function in another illustrativeembodiment.

FIG. 10 is a flowchart illustrating a method for determining to applysensor data to a machine learning function in yet another illustrativeembodiment.

FIG. 11 is a flowchart illustrating a method for determining to applysensor data to a machine learning function in still another illustrativeembodiment.

FIG. 12 illustrates an environment for capturing training data with auser device in an illustrative embodiment.

DESCRIPTION

The figures and the following description illustrate specific exampleembodiments of the disclosure. It will thus be appreciated that thoseskilled in the art will be able to devise various arrangements that,although not explicitly described or shown herein, embody the principlesof the disclosure and are included within the scope of the disclosure.Furthermore, any examples described herein are intended to aid inunderstanding the principles of the disclosure and are to be construedas being without limitation to such specifically recited examples andconditions. As a result, the disclosure is not limited to the specificembodiments or examples described below, but by the claims and theirequivalents.

FIG. 1 illustrates a worksite environment 100 in an illustrativeembodiment. The worksite environment 100 is enhanced with a worksitesafety system 110 configured to detect and prevent potentially unsafeconditions in the worksite environment 100. To improve its ability toautomatically detect and respond to safety issues, the worksite safetysystem 110 may receive input from user equipment 120 _(1 . . . N)operated by respective trainees 122 _(1 . . . N) taking safety coursesregarding the worksite environment 100. The trainees 122 may beemployees or technicians of an organization or company that implementsmandatory and/or periodic safety training courses to improve safety ofthe worksite environment 100.

The worksite safety system 110 is configured to refine the assessmentsprovided by the trainees 122 regarding various scenes or scenarios inthe worksite environment 100, and to train a machine learning function116 using the refined data. More particularly, the trainees 122 areprompted to provide answers regarding safety scenarios of the worksiteenvironment 100 as part of a safety course or curriculum content. Theworksite safety system 110 performs a series of steps to establish whichof the trainees 122 have a relatively higher level of trust in correctlyassessing the level of risk in the worksite environment 100.

By processing answers provided by trainees 122 having relatively higherlevels of trust, the worksite safety system 110 is configured to minedata for previously unknown or unascertained characteristics of theworksite environment 100 that may pose safety issues. Advantageously,applying high-trust data as input to the machine learning function 116results in highly accurate automatic detection and intelligent responseto safety issues in the worksite environment 100. Additionally, therefined data sets are collected in the course of administrating worksitesafety courses, thereby eliminating the cost of hiring specialists tocollect, filter, and classify enough training data to be used for usingmachine learning to improve safety in the worksite environment 100.Still further, in the process of collecting refined data sets, theworksite safety systems 110 helps identify better, more granularworksite examples for humans to assess and refine, thus providingautomatic, continuous updates to the safety curriculum content itselffor comprehensive, up-to-date safety training and enhanced worksitesafety.

The worksite safety system 110 includes an interface 112, one or moreprocessor(s) 114, and memory 118. The interface 112 may comprise acomponent or device (e.g., transceiver, antenna, etc.) configured toperform wired and/or wireless communication over a network with othersystems and devices. The processor 114 represents the internalcircuitry, logic, hardware, etc., that provides the functions of theworksite safety system 110. The memory 118 may comprise a computerreadable storage medium for data, instructions, applications, etc.,accessible by the processor 114.

The processor 114 may implement the machine learning function 116. Themachine learning function 116 may be implemented in any combination ofhardware, firmware, and/or software operable to implement machinelearning techniques. Machine learning generally refers to an automatedprocess capable of parsing input data, learning from the data, and thenadapting its output based on its learning. This differs from traditionalcomputer processes where instructions or programming is predefined andexplicit such that the same steps are repeated given the same input.That is, rather than having activities defined in advance, the machinelearning function 116 may be trained to observe patterns in data andadaptively adjust actions or steps to take over time without explicithand-coded programming or user intervention/instruction.

The worksite safety system 110 may be communication with one or morecamera(s) 130, one or more sensor(s) 132, and a response alert system134. The camera(s) 130 are generally positioned to have a field of viewof the worksite environment 100 to capture visual/audio data of one ormore scenes. In this example, the camera 130 records a first scene 141and a second scene 142 of a technician 150 and machine 160 in theworksite environment 100. As will be described in greater detail below,the first scene 141 may depict a known hazard (e.g., technician is notwearing head protection near the machine 160), and the second scene 142may depict an unknown hazard (e.g., technician is wearing helmet 154 butother safety issues are possible). The first scene 141 and second scene142 may represent images captured by camera(s) 130 at different timesand/or at different locations in the worksite environment 100.

The sensor(s) 132 may additionally or alternatively provide data for thefirst scene 141 and/or second scene 142. The sensor 132 may include anynumber of devices configured to capture other types of environmentaldata of the worksite environment 100. Examples of environmental datainclude, but are not limited to, temperature, humidity, barometricpressure, pollution or chemical levels, relative locations oftechnicians 150 and machines 160, etc. Data captured by camera(s) 130and sensor(s) 132 may be transmitted to the worksite safety system 110via interface 112.

Additionally, the technician 150 may wear a wearable sensor 152 thatincludes any device configured to capture/transmit biological data ofthe technician 150 for the worksite safety system 110. Examples ofbiological data include, but are not limited to, heart rate,respiration, blood pressure, body temperature, etc. Alternatively oradditionally, the wearable sensor 152 may include tactile feedbackarticles (e.g., gloves) configured to detect and report touch/pressuredata experienced by the technician 150. Similarly, the machine 160 mayinclude a machine sensor 162 that includes any device configured tocapture/transmit machine data of the machine 160 for the worksite safetysystem 110. Examples of machine data include, but are not limited to,revolutions per minute, hydraulic pressure, time to pressurize, forceapplied, machine cycle, amps drawn, etc.

The response alert system 134 includes any system configured to respondto a detected safety event. Example functions of the response alertsystem 134 includes, but is not limited to, calling emergency services,cutting power to the machine 160, highlighting (e.g., via a display orlaser pointer) a location of the safety issue in the worksiteenvironment 100, and/or sounding an alarm. Alternatively oradditionally, the response alert system 134 may provide appropriatecontextual information for personnel responding to the safety issue suchas alerting those in the area to a location of nearby safety equipmentrelated to the event (e.g., location of defibrillators, a type of workglove, tool for the machine 160, etc.).

It will be appreciated that the worksite environment 100 is an exampleenvironment for discussion purposes and that the features of theworksite safety system 110 described herein may be employed inalternative environments and applications. Illustrative examples anddetails of the operation of the worksite safety system 110 are discussedbelow.

FIG. 2 is a flowchart illustrating a method 200 of refining trainingdata for a machine learning function to enhance worksite safety in anillustrative embodiment. The methods herein may be described withrespect to the worksite environment 100 and worksite safety system 110of FIG. 1, although one skilled in the art will recognize that themethods may be performed with other environments and systems. The stepsof the methods described herein are not all inclusive, may include othersteps not shown, and may also be performed in alternative orders.

In step 202, the worksite safety system 110 receives first evaluationsof a first scene 141 from a group of the trainees 122, the first scene141 belonging to safety curriculum content and depicting a worksite witha known hazard that is associated in memory 118 with a hazard profile.In one embodiment, the first scene 141 of the safety curriculum contentincludes video content or image data recorded by one or more cameras130. Alternatively or additionally, the first scene 141 may compriseother types of sensor data such as sound or various environmentalmetrics available via sensors 132, wearable sensors 152, and/or machinesensors 162. The first scene 141 may be displayed on respective userdevices 120 or in a centralized setting such as a classroom.

In step 204, the worksite safety system 110 determines a trustedsubgroup of the trainees 122 that correctly identified the known hazardin the first scene based on a match between the first evaluations andthe hazard profile. The hazard profile may include data describing theascertained characteristics (e.g., image data, sensor data, etc.) of theknown hazard. A known hazard may have been previously established orascertained as hazardous by expert analysis or previous traineeevaluations. Thus, the worksite safety system 110 may establish which ofthe trainees 122 have a threshold level of trust based on their correctidentification of hazardous characteristics as defined in the predefinedhazard profile. An evaluation or identification of hazardouscharacteristics may include any number of response formats includingtext, audio, multiple choice, interactive selection of objects on thedisplay, etc. that may be matched with data in a hazard profile of aknown hazard.

The trusted subgroup may be established by processing multiple rounds ofscenes with known hazards and corresponding evaluations. That is, theworksite safety system 110 may determine which of the trainees 122 haverelatively higher levels of trust after processing multiple rounds ofanswers provided by the trainees 122 to various first scenes 141depicting various known hazards. Thus, it may be understood that term“first” in first scene 141, first evaluations, etc. may refer to arelation to a known hazard rather than to a single instance ofsuccessful training.

In step 206, the worksite safety system 110 receives second evaluationsof a second scene 142 from the trusted subgroup of the trainees, thesecond scene 142 depicting the worksite with an unknown hazard. Theunknown hazard, unlike the known hazard, is a scenario or situation inthe worksite environment 100 which has not yet been definitivelyestablished as hazardous or non-hazardous. For example, the unknownhazard may be a condition not previously analyzed by expert analysts ortrainees 122 as being hazardous. Alternatively or additionally, theunknown hazard may depict an ambiguous or “gray area” edge case forwhich different trainees 122 or even expert analysists may disagreecontains a potentially unsafe condition. Embodiments for selecting orgenerating the unknown hazard to present to the trusted subgroup of thetrainees will be later described. In some embodiments, the worksitesafety system 110 receives multiple rounds of answers provided by thetrusted subgroup to various second scenes 142 depicting various unknownhazards. Thus, it may be understood that term “second” in second scene142, second evaluations, etc. may refer to a relation to an unknownhazard rather than to a second single instance of training.

In step 208, the worksite safety system 110 trains the machine learningfunction 116 based on the second evaluations from the trusted subgroupof the trainees 122 for automatic identification of hazard indicationsin the second scene 142 depicting the worksite with the unknown hazard.By training the machine learning function 116 with data provided bytrainees 122 having relatively higher levels of trust, the method 200improves the fidelity of the training set while teaching/training theindividual trainees 122. This leverages an organization's safetytraining curriculum process for their personnel to generate largeamounts of refined data training input for machine learning to improvesafety in the worksite environment 100 without the cost of hiringspecialists to collect, filter, and classify the training data.

FIG. 3 is a flowchart illustrating a method 300 of applying refinedtraining data to a machine learning function for worksite safety in anillustrative embodiment. In step 302, the one or more camera(s) 130and/or one or more sensor(s) 132 monitor the worksite environment 100 torecord sensor data. Sensor data may therefore include image/video dataobtained via camera(s) 130 and/or other types of sensor data obtainedvia sensor(s) 132, wearable sensors 152, and/or machine sensors 162 aspreviously described.

In step 304, the worksite safety system 110 applies the sensor data tothe machine learning function 116 trained with the second evaluations toautomatically detect the hazard indications of the unknown hazard in thesensor data. And, in step 306, in response to detecting the hazardindications of the unknown hazard in the sensor data, the worksitesafety system 110 generates an automatic action to perform forresponding to the unknown hazard. Thus, the worksite safety system 110uses the high-trust input of trainees 122 to improve its machinelearning output in automatically detecting and responding to safetyhazards and risks in the worksite environment 100 in real time.

In step 308, the worksite safety system 110 determines whether thedetection of the hazard and action(s) taken in response were correct.For example, the worksite safety system 110 may receive input regardingthe effectiveness of its automatically generated machine learningoutput. If the hazard detection and response is correct, the method 300may return to continuously perform steps 302-306 for detecting andresponding to potential hazards in the worksite environment 100.Otherwise, if the hazard detection and response is inadequate, themethod 300 may proceed to step 310 where the worksite safety system 110receives feedback for improving its training data collection and/orsafety curriculum content.

FIG. 4 is a flowchart illustrating a method 400 of refining trainingdata for a machine learning function to enhance worksite safety inanother illustrative embodiment. In step 402, the worksite safety system110 receives input identifying a deficiency of the safety curriculumcontent based on one or more of the first evaluations, secondsevaluations, and feedback of hazard identification and response. In step404, the worksite safety system 110 generates additional hazard databased on received input. Step 404 may include one or more of steps406-410 described below.

In step 406, the worksite safety system 110 determines a subset of thetrusted subgroup of the trainees 122 that provided an evaluation thatincludes additional hazard data. Such additional hazard data may beobtained, for example, in step 408 in which the trusted subgroup oftrainees 122 are requested to provide at least one characteristic ofanother scene in the safety curriculum content and/or worksiteenvironment 100. Alternatively or additionally, in step 410, theworksite safety system 110 may identify additional hazard by analyzingthe evaluation of a trainee to determine at least one alternativecharacteristic of the first scene 141 that does not match the hazardprofile. In this way, the worksite safety system 110 may use evaluationanswers provided by one or more trainees 122 that have been establishedas consistently providing accurate input to mine for additional data inthe safety curriculum content and/or worksite environment 100 that hasnot yet been ascertained.

In step 412, the worksite safety system 110 generates the second scene142 depicting the unknown hazard. Step 412 may include one or more ofsteps 414-416 described below. In step 414, the second scene 142 may becreated based on the additional hazard data. For instance, afterobtaining additional hazard data from one or more trusted trainees 122,the worksite safety system 110 may insert the characteristics of theadditional hazard data into a new scenario to create the second scene142. Alternatively or additionally, the second scene 142 may begenerated by retrieving scene data that preceded the first scene 141 andthe known hazard. For example, the worksite safety system 110 mayautomatically retrieve image/sensor data that preceded a known hazard tomine for precursors or an unknown hazard leading up to a hazardousevent.

FIG. 5 is a flowchart illustrating a method 500 of modifying safetycurriculum content in an illustrative embodiment. In step 502, thesecond scene 142 depicting the unknown hazard is displayed to thetrusted subgroup of the trainees 122. And, in step 504, the worksitesafety system 110 receives second evaluations of the second scene 142from the trusted subgroup of the trainees 122, wherein the second scene142 depicts the worksite with the unknown hazard.

In step 508, the worksite safety system 110 determines whether to modifythe safety curriculum content. For example, the worksite safety system110 may determine based on the second evaluations (e.g., high-trustfeedback related to an unknown hazard) that the safety curriculumcontent may be improved by being changed to include a safety trainingexample or scenario that covers the second scene 142. Over time, if athreshold level of certainty is established for a second scene 142(e.g., by calculating a rating score based on weighted user trust and/oroverlapping agreement among trainee feedback), the worksite safetysystem 110 may convert the second scene 142 into a first scene 141 andestablish a hazard profile for it that becomes part of the standardcourse content used to train future trainees.

If, in step 508, it is determined to modify the safety curriculumcontent, the method 500 may return to step 202 of method 200 to continueto refine machine learning input using the safety curriculum content.Otherwise, if in step 508, it is determined to not modify the safetycurriculum content, the method 500 may return to step 302 of method 300to continue to automatically detect and respond to potential hazards inthe worksite environment 100 using inputs via the current version of thesafety curriculum content. Thus, methods 200-500 provide a technicalbenefit in continuously adapting safety training courses and machinelearning input in an interrelated manner.

FIG. 6 is a block diagram of a training system 600 in an illustrativeembodiment. The training system 600 includes a training controller 610that implements or interfaces with one or more components of theworksite safety system 110 to perform machine learning techniques forautomation at a worksite. For example, the worksite safety system 110may include one or more of a safety curriculum system 616, activitymonitoring system 618 (e.g., cameras 130, sensors 132, wearable sensors152, etc.), and response alert system 134 to implement functionspreviously described.

The training controller 610 may additionally or alternatively implementor interface with a gamification system 620 configured to incentivizeuser input and classification of training input 612 to be applied to themachine learning function 116. The gamification system 620 stores userprofiles 622 that tracks, for individual users or players, userperformance ratings 624 and user confidence levels 626 in accuratelycapturing training data for machine learning applications. As describedin greater detail below, the gamification system 620 may manage apoint-based system that encourages targeted data collection to use asrefined input for machine learning. Additionally, the gamificationsystem 620 may manage groups of users within knowledge domains 628 thatcoordinates training data collection according to an area of expertiseto which users may belong. For example, a user or employee belonging toan engineering group within an organization may have increased weightapplied for collected data related to their engineering expertise.

The training controller 610 may receive sensor data input from anindividual 604 via a dashboard 602 operated by an employee or supervisorof an organization. The dashboard 602 may include an interface 606and/or graphical user interface (GUI) 608 to communicate and displaydata. The operator may initialize the training system 600 by connectingwith the training controller 610 and accessing baseline data. Encryptedkeys along with baseline data may be sent to a cloud aggregator 640 thatunlocks and requests profiled data, aggregates data, and generates thelatest course or game materials. For example, the safety curriculumsystem 616 and/or gamification system 620 may generate data capturinginstructions for the user based on their profile data. Baseline data foran individual user may be established based on their user profile 622and data from other similar profiles (e.g., users belonging to the sameknowledge domain 628).

After sending data capturing instructions for the user to classify ascenario or object in an environment, the training controller 610receives the captured data and feeds it back to the cloud aggregator 640to be observed, sorted, and tagged. An Artificial Intelligence (AI)engine 642 in the cloud aggregator 640 updates logic tree changes toreflect learnings based on human evaluation of the captured data. The AIengine 642 may compare its own response to that of humans averaged andweighted across trainees to simultaneously improve fidelity of atraining set while administering course material as previouslydescribed. A training database 650 stores collected data for analyzingpatterns for continuously improving performance. The training database650 may store video, image, audio, or other type of sensor data forenhancing training data sets as deficiencies arise or enhancements areidentified. For example, an end user may be directed, as a part of theirsafety training, to go and find potential examples of a given trainingscenario or environmental condition that may be fed back to the trainingcontroller 610 for rating and processing and incorporated into thesafety curriculum system 616 via the cloud aggregator 640 and trainingdatabase 650. Sensor data sent to the training controller 610 isaggregated and sorted with other recorded data to continuously build andrefine course and game materials. Game or course material may beoptimized relative to needs of a particular role in an environment andcompared with like populations. Adjustments to improve a game or coursemay be made and sent to users as future training is requested. Theresponse alert system 134 analyzes actions taken based on escalationlevel of the event at hand to learn what actions to take.

FIG. 7 is a flowchart illustrating a method 700 of collecting trainingdata for a machine learning function to enhance worksite operations inan illustrative embodiment. The methods herein may be described withrespect to the training system 600, although one skilled in the art willrecognize that the methods may be performed with other environments andsystems. In step 702, the training controller 610 instructs a user, tocapture, via user device, first sensor data of an object performing oneor more known actions. The user device may include, for example,personal computers, laptops, smartphones, or other devices/sensors.Known actions may have been previously established or ascertained byexpert analysis or previous user/player input.

In step 704, the training controller 610 receives (e.g., via interface606) the first sensor data captured by the user device. In step 706, thetraining controller 610 determines whether the first sensor datacaptured by the user device correctly depicts the object performing theone or more known actions. If the first sensor data does not correctlydepict the known action, the method 700 may return to step 702 toperform another round of collecting sensor data related to knownactions. Otherwise, the method 700 may proceed to step 708.

In step 708, the training controller 610 allocates points as an award tothe user of the user device for correctly capturing the objectperforming the one or more known actions. The awarded points may beassociated with the user profile 622 or user performance ratings 624 inthe gamification system 620 and be visually displayed to the user viathe user device. Points may be awarded based on relative difficulty ofcapturing the object performing a target action and provide a basis ofcompetition among a population of users. In this way, a population ofusers are incentivized to provide large amounts of refined data traininginput for machine learning to improve worksite operations without thecost of hiring specialists to collect, filter, and classify the trainingdata.

Thus, the training system 600 may implement a point collection system totrack use achievement among a population or sub-population of users. Forexample, points may be awarded to users to achieve training credit, earncredentials or badges, rank amongst peers of a group (e.g., engineeringteam), provide geo-temporal incentives and/or domain-specific incentivesfor increased alertness for specific domains, and/or achieve specificdata collection from a targeted user base. The incentives may be drivenby backend machine learning training model gaps or fidelityimprovements. Thus, the training system 600 is able to incentivize endusers to acquire, identify, and provide machine learning training data.

In step 710, the training controller 610 determines whether the user ofthe user device has exceeded a threshold of awarded points. If the userhas not exceeded the point threshold, the method 700 may return to step702 to perform additional rounds of collecting sensor data related toknown actions. Otherwise, if the user has exceeded the point threshold,the method 700 proceeds to step 712.

In step 712, the training controller 610 instructs the user to capture,via the user device, second sensor data of the object performing one ormore unknown actions. Unlike a known action, an unknown action maycomprise an action performed by the object which has not yet beendefinitively established by the training system 600. For example, theunknown action may be an action not previously established by expertanalysts or a user population. Alternatively or additionally, theunknown action may depict an ambiguous or “gray area” edge case forwhich different players or expert analysists may disagree constitutesbehavior that defines the target action.

In step 714, the training controller 610 receives (e.g., via interface606) the second sensor data captured by the user device. And, in step716, the training controller 610 trains the machine learning function116 based on the second sensor data of the object performing the one ormore unknown actions as identified by the user having exceeded thethreshold of awarded points. Thus, the method 700 improves the fidelityof the training set by incentivizing users to collect and classifytraining data with a point-based game. Additionally, the training setscollected and applied to machine learning are advantageously refined inthe gamification system 620 by using inputs from players that have beenqualified in the game as having a level of experience in identifyingtarget actions that occur in an environment.

FIG. 8 is a flowchart illustrating a method 800 for determining to applysensor data to a machine learning function in an illustrativeembodiment. In step 802, the training controller 610 calculates a ratingof the user in correctly capturing the object performing the one or moreknown actions in comparison to other users. In step 804, the trainingcontroller 610 generates an instruction for the user to capture thesecond sensor data based on the rating of the user. For example, in oneembodiment, the instructions request the user to capture the secondsensor data with a target sensor. And, in step 806, the trainingcontroller 610 determines to collect the second sensor data and applythe second sensor data to the machine learning function 116 based on therating of the user. For example, for a user with a high rating (e.g.,cumulative points exceed a threshold), the training controller 610 mayapply increased weighting to classification/input from that user forgenerating machine learning training data to automatically detectunknown or undefined action that occurs in an environment. Conversely,users with relatively low ratings may have decreased weighting or inputthat is discarded for generating machine learning training data.Advantageously, training data is refined by the gamification and ratingprocess.

FIG. 9 is a flowchart illustrating a method 900 for determining to applysensor data to a machine learning function in another illustrativeembodiment. In step 902, the training controller 610 receives (e.g., viainterface 606) a user confidence level of correctly assessing the one ormore unknown actions of the object. And, in step 904, the trainingcontroller 610 determines to collect the second sensor data and applythe second sensor data to the machine learning function 116 based on theuser confidence level of the user. In some embodiments, the trainingcontroller 610 determines whether to collect/apply a user's secondsensor data submission based on a combination of the rating and the userconfidence level of the user. Thus, machine learning training data isrefined taking into account relative levels of user confidence incorrectly classifying data through a gamification technique.

FIG. 10 is a flowchart illustrating a method 1000 for determining toapply sensor data to a machine learning function in yet anotherillustrative embodiment. In step 1002, the training controller 610determines whether the user belongs to an expert user domain associatedwith correctly capturing the object performing the one or more knownactions. And, in step 1004, the training controller 610 determines tocollect the second sensor data and apply the second sensor data to themachine learning function 116 based on the user belonging to the expertuser domain. Advantageously, machine learning training data may berefined according to various ability levels of a user in classifying aparticular scenario.

FIG. 11 is a flowchart illustrating a method 1100 for determining toapply sensor data to a machine learning function in still anotherillustrative embodiment. In step 1102, the training controller 610determines a correlation between a characteristic of acquiring the firstsensor data and performance of the user in correctly capturing theobject performing the one or more known actions. For example, thecharacteristic may include a location of acquiring the first sensordata, a timing of acquiring the first sensor data, and/or a sensor typeused to collect the first sensor data. And, in step 1104, the trainingcontroller 610 determines to collect the second sensor data and applythe second sensor data to the machine learning function 116 based on thecorrelation. Therefore, machine learning training data may be refined byapplying weights to user input based on correlating factors that tend tocorroborate the user's classification as having increased or decreasedlikelihood of accuracy.

FIG. 12 illustrates an environment 1200 for capturing training data witha user device 1202 in an illustrative embodiment. The user device 1202may include a smartphone with hardware components 1204 such a display1206 operable to receive touch screen input and a speaker 1208 toproject sound. The user device 1202 may also include sensor components1210 such as an environment sensor 1212 to capture one or morecharacteristics of the environment 1200, microphone 1214 to captureaudio data, and camera 1216 to capture image data and/or video data.

In this example, the camera 1216 captures an object 1250 (e.g., aircraftmarshaller) performing a target action 1252 (e.g., signaling an aircraftto turn left). Data captured by the sensor components 1210 may be storedin memory 1240 of the user device 1202 as first sensor data 1241 orsecond sensor data 1242. For instance, instead of imagery, the userdevice 1202 may collect audio of machinery in attempt to capture aparticular target action (e.g., operating with bearing failure,excessive cutter wear, etc.). Alternatively or additionally, theenvironment sensor 1212 may include a vibration sensor to collectvibration data (e.g., captured by resting the user device 1202 on amachine) or a thermal sensor to collect thermal data (e.g., for dentdetection in a fuselage of an aircraft).

The memory 1240 may further store sensor acquisition parameters 1243configured to correlate the collection of sensor data with variouscharacteristics. The sensor acquisition parameters 1243 may facilitate acorrelation that is based on the location of sensor acquisition, thetime of sensor acquisition (e.g., time of day relative to breaks,arrivals, departures, etc. which indicate periods of user fatigue orpreoccupation), and/or the type of sensor acquisition (e.g., image,audio, thermal, etc.). The training system 600 and/or user device 1202may correlate the sensor acquisition parameters 1243 with user trust,performance, or reliability in correctly capturing the target action1252.

The user device 1202 may also implement a gamification application 1244configured to incentivize sensor data collection for the training system600. The gamification application 1244 may communicate with the trainingsystem 600 via cloud 1260 to implement a gamification system for apopulation of users across multiple servers over the Internet and/orother networks. That is, gamification for the machine learning function116 may be implemented across a platform that is both tailored to theuser device 1202 and yet common to a plurality of user devices managedby the training system 600. As an example, the gamification application1244 may generate, or receive from the training system 600, captureinstructions 1245 tailored to a performance history of the user. Forinstance, the user may be provided with a query that at attempts toinfluence acquisition choices (e.g., presented with a series of sensordata acquisitions with a targeted sensor and domain such as image,sound, etc.). The tailored capture instructions 1245 may be used toassess user sway and correlate sway relative to areas of potentialambiguity (e.g., location, time of day, domain type, etc.).

The gamification application 1244 may also prompt for, or receive, auser confidence level 1246 of the acquired sensor data and/or sensordomain. For example, the user device 1202 may display a slider bar thatis selectable between zero and one hundred percent. The user confidencelevel 1246 may be adjusted based on the user's historical performance,expertise domain, and or preference for the sensor data and domain underacquisition. If user confidence is low or below a threshold, thegamification application 1244 may provide a targeted comparison ofexisting sensor data that allows a user to perform a comparison. If userconfidence is consistently low, the gamification application 1244 maypresent the user with three or more options, wherein at least one optionincludes the desired sensor signal and at least one other optionincludes the desired sensor signal in a slightly altered way.

The gamification application 1244 may provide user anonymity for thetraining system 600 where each user is conferred trust based on merit.This supports expansion into new areas where a user may be willing totake increased risks in incorrectly classifying information in theenvironment 1200. Alternatively or additionally, a user and associatedcollection profile may be published within a community or knowledgedomain to establish performance-based brand or status. This supportsloss aversion and encourages users to be careful about theirclassifications of the environment 1200. To improve user performancewith positive reinforcement, the gamification application 1244 mayprovide an initial test, and then reinforce successful choices byshowing evidence of reasons they might have made a particularclassification choice.

To communicate with the training system 600, the user device 1202 mayinclude network components 1230 such as a cellular communicationinterface 1232, global positioning satellite (GPS) 1234, WiFi 1236, andBluetooth 1238. The user device 1202 also includes hardware 1220 such asone or more processor(s) 1224 configured to process computer-executableinstructions to control the operation of the user device 1202,Input/Output (I/O) peripherals 1222 such as keyboards and externalstorage/sensor devices, and random access memory (RAM) 1226. Thoughsteps of the flowcharts herein generally refer to the training system600, it is understood that steps may be performed at the user device1202 and training system 600 in various combinations. The machinelearning function 116 may implement any number of suitable machinelearning processes, algorithms, or techniques including anomalydetection, Naive Bayes classifiers, support vector machines, decisiontree learning, neural network learning, reinforcement learning, etc. Themachine learning processes may be tailored with machine learningparameters by matter of design choice (e.g., kernel type for supportvector machines, number of trees for decision trees, etc.).

The machine learning function 116 may transform the input values todetermine patterns, correlations, features, statistics, predictions, orclassifications. The machine learning function 116 may monitor sensordata record in memory 1240 of the user device 1202 to establish userprofiles and perform automated actions via machine learning output. Thatis, the training system 600 and/or user device 1202 may receive anoutput from the machine learning function 116 to help automate actionsin the environment. In this example, user-classified data related to anaircraft marshaller signaling an aircraft may be used to automaticallydetect/prevent a hazard in the environment 1200 related to a missteptaken by the pilot or aircraft marshaller.

Any of the various control elements (e.g., electrical or electroniccomponents) shown in the figures or described herein may be implementedas hardware, a processor implementing software, a processor implementingfirmware, or some combination of these. For example, an element may beimplemented as dedicated hardware. Dedicated hardware elements may bereferred to as “processors”, “controllers”, or some similar terminology.When provided by a processor, the functions may be provided by a singlededicated processor, by a single shared processor, or by a plurality ofindividual processors, some of which may be shared. Moreover, explicituse of the term “processor” or “controller” should not be construed torefer exclusively to hardware capable of executing software, and mayimplicitly include, without limitation, digital signal processor (DSP)hardware, a network processor, application specific integrated circuit(ASIC) or other circuitry, field programmable gate array (FPGA), readonly memory (ROM) for storing software, random access memory (RAM),non-volatile storage, logic, or some other physical hardware componentor module.

Also, a control element may be implemented as instructions executable bya processor or a computer to perform the functions of the element. Someexamples of instructions are software, program code, and firmware. Theinstructions are operational when executed by the processor to directthe processor to perform the functions of the element. The instructionsmay be stored on storage devices that are readable by the processor.Some examples of the storage devices are digital or solid-statememories, magnetic storage media such as a magnetic disks and magnetictapes, hard drives, or optically readable digital data storage media.

Although specific embodiments are described herein, the scope of thedisclosure is not limited to those specific embodiments. The scope ofthe disclosure is defined by the following claims and any equivalentsthereof.

What is claimed is:
 1. A worksite safety system comprising: an interfaceconfigured to receive first evaluations of a first scene from a group oftrainees, the first scene belonging to safety curriculum content anddepicting a worksite with a known hazard that is associated in memorywith a hazard profile; and a safety training controller configured todetermine a trusted subgroup of the trainees that correctly identifiedthe known hazard in the first scene based on a match between the firstevaluations and the hazard profile, the interface configured to receivesecond evaluations of a second scene from the trusted subgroup of thetrainees, the second scene depicting the worksite with an unknownhazard, and the safety training controller configured to train a machinelearning function based on the second evaluations from the trustedsubgroup of the trainees for automatic identification of hazardindications in the second scene depicting the worksite with the unknownhazard.
 2. The worksite safety system of claim 1 further comprising: oneor more sensors configured to record sensor data of the worksite, thesafety training controller configured to apply the sensor data to themachine learning function trained with the second evaluations toautomatically detect the hazard indications of the unknown hazard in thesensor data.
 3. The worksite safety system of claim 2 wherein: thesafety training controller configured, in response to detecting thehazard indications of the unknown hazard in the sensor data, to generatean automatic action to perform for responding to the unknown hazard. 4.The worksite safety system of claim 1 wherein: the safety trainingcontroller configured to modify the safety curriculum content based onthe first evaluations and the second evaluations.
 5. The worksite safetysystem of claim 1 wherein: the safety training controller configured todetermine a subset of trainees of the trusted subgroup that provided anevaluation that includes additional hazard data, and to create thesecond scene based on the additional hazard data.
 6. The worksite safetysystem of claim 1 wherein: the safety training controller configured togenerate the second scene by retrieving scene data that preceded thefirst scene and the known hazard.
 7. A method of training a machinelearning function for worksite safety, the method comprising: receivingfirst evaluations of a first scene from a group of the trainees, thefirst scene belonging to safety curriculum content and depicting aworksite with a known hazard that is associated in memory with a hazardprofile; determining a trusted subgroup of the trainees that correctlyidentified the known hazard in the first scene based on a match betweenthe first evaluations and the hazard profile; receiving secondevaluations of a second scene from the trusted subgroup of the trainees,the second scene depicting the worksite with an unknown hazard; andtraining the machine learning function based on the second evaluationsfrom the trusted subgroup of the trainees for automatic identificationof hazard indications in the second scene depicting the worksite withthe unknown hazard.
 8. The method of claim 7 further comprising:monitoring the worksite with one or more sensors to record sensor dataof the worksite; and applying the sensor data to the machine learningfunction trained with the second evaluations to automatically detect thehazard indications of the unknown hazard in the sensor data.
 9. Themethod of claim 8 further comprising: in response to detecting thehazard indications of the unknown hazard in the sensor data, generatingan automatic action to perform for responding to the unknown hazard. 10.The method of claim 9 wherein: the automatic action includes one or moreof: generating a warning of the hazard indications present in theworksite, and automatically powering down a machine in the worksite. 11.The method of claim 7 further comprising: modifying the safetycurriculum content based on the first evaluations and the secondevaluations.
 12. The method of claim 7 further comprising: determining asubset of trainees of the trusted subgroup that provided an evaluationthat includes additional hazard data; and creating the second scenebased on the additional hazard data.
 13. The method of claim 12 furthercomprising: identifying the additional hazard data by: analyzing theevaluation of a trainee to determine at least one alternativecharacteristic of the first scene that does not match the hazardprofile.
 14. The method of claim 12 further comprising: generating theadditional hazard data by: requesting the trusted subgroup of thetrainees to provide at least one characteristic of another scene in thesafety curriculum content; receiving the at least one characteristic ofthe another scene; and identifying the at least one characteristic asthe additional hazard data.
 15. The method of claim 7 furthercomprising: generating the second scene by retrieving scene data thatpreceded the first scene and the known hazard.
 16. A non-transitorycomputer readable medium embodying programmed instructions executed by aprocessor, wherein the instructions direct the processor to perform amethod of training a machine learning function for worksite safety, themethod comprising: receiving first evaluations of a first scene from agroup of the trainees, the first scene belonging to safety curriculumcontent and depicting a worksite with a known hazard that is associatedin memory with a hazard profile; determining a trusted subgroup of thetrainees that correctly identified the known hazard in the first scenebased on a match between the first evaluations and the hazard profile;receiving second evaluations of a second scene from the trusted subgroupof the trainees, the second scene depicting the worksite with an unknownhazard; and training the machine learning function based on the secondevaluations from the trusted subgroup of the trainees for automaticidentification of hazard indications in the second scene depicting theworksite with the unknown hazard.
 17. The computer readable medium ofclaim 16 wherein the method further comprises: monitoring the worksitewith one or more sensors to record sensor data of the worksite; andapplying the sensor data to the machine learning function trained withthe second evaluations to automatically detect the hazard indications ofthe unknown hazard in the sensor data.
 18. The computer readable mediumof claim 17 wherein the method further comprises: in response todetecting the hazard indications of the unknown hazard in the sensordata, generating an automatic action to perform for responding to theunknown hazard.
 19. The computer readable medium of claim 16 wherein themethod further comprises: modifying the safety curriculum content basedon the first evaluations and the second evaluations.
 20. The computerreadable medium of claim 16 wherein the method further comprises:determining a subset of trainees of the trusted subgroup that providedan evaluation that includes additional hazard data; and creating thesecond scene based on the additional hazard data.